Setup
library(here)
here() starts at /Users/maarten/Documents/projects/rof-embeddings
library(data.table)
Registered S3 method overwritten by 'data.table':
method from
print.data.table
data.table 1.14.8 using 1 threads (see ?getDTthreads). Latest news: r-datatable.com
**********
This installation of data.table has not detected OpenMP support. It should still work but in single-threaded mode.
This is a Mac. Please read https://mac.r-project.org/openmp/. Please engage with Apple and ask them for support. Check r-datatable.com for updates, and our Mac instructions here: https://github.com/Rdatatable/data.table/wiki/Installation. After several years of many reports of installation problems on Mac, it's time to gingerly point out that there have been no similar problems on Windows or Linux.
**********
library(fst)
fst package v0.9.8
library(text)
This is text (version 0.9.99.2).
Text is new and still rapidly improving.
Newer versions may have improved functions and updated defaults to reflect current understandings of the state-of-the-art.
Please send us feedback based on your experience.
Please note that defaults has changed in the textEmbed-functions since last version; see help(textEmbed) or www.r-text.org for more details.
library(stringr)
library(ggplot2)
library(scico)
library(plotly)
Attaching package: ‘plotly’
The following object is masked from ‘package:ggplot2’:
last_plot
The following object is masked from ‘package:stats’:
filter
The following object is masked from ‘package:graphics’:
layout
library(uwot)
Loading required package: Matrix
library(extrafont)
Registering fonts with R
set.seed(42)
theme_set(theme_bw(base_size = 16, base_family = "sans"))
# Run once to import the Nunito font
# font_import(pattern = "Nunito")
# loadfonts()
Data
Load rate of forgetting predictions per fact (from the Fact
prediction method in https://doi.org/10.31234/osf.io/z3vtn),
as well as the word or phrase associated with each fact:
# English
rof_pred_en <- read_fst(here("data", "pred_fact_Stepping_Stones.fst"), as.data.table = TRUE)
fstcore package v0.9.14
(OpenMP was not detected, using single threaded mode)
facts_en <- read_fst(here("data", "answers_Stepping_Stones2.fst"), as.data.table = TRUE)
setnames(facts_en, "fact_id_uniq_merged", "fact_id")
fact_rof_en <- rof_pred_en[facts_en, on = "fact_id"]
fact_rof_en <- fact_rof_en[!is.na(mu)]
fact_rof_en <- fact_rof_en[, .(fact_id,
n_obs = kappa - 1,
rof = mu,
answer = answer)]
fact_rof_en[, answer_language := ifelse(tstrsplit(fact_id, "_", fixed = TRUE, keep = 3L)[[1]] == "1", "NL", "EN")]
fact_rof_en[, course := "English"]
# French
rof_pred_fr <- read_fst(file.path("data", "pred_fact_Grandes_Lignes.fst"), as.data.table = TRUE)
facts_fr <- read_fst(file.path("data", "answers_Grandes_Lignes2.fst"), as.data.table = TRUE)
setnames(facts_fr, "fact_id_uniq_merged", "fact_id")
fact_rof_fr <- rof_pred_fr[facts_fr, on = "fact_id"]
fact_rof_fr <- fact_rof_fr[!is.na(mu)]
fact_rof_fr <- fact_rof_fr[, .(fact_id,
n_obs = kappa - 1,
rof = mu,
answer = answer)]
fact_rof_fr[, answer_language := ifelse(tstrsplit(fact_id, "_", fixed = TRUE, keep = 3L)[[1]] == "1", "NL", "FR")]
fact_rof_fr[, course := "French"]
# Combine into a single data.table
fact_rof <- rbind(fact_rof_en, fact_rof_fr)
Some answers occur across multiple facts (e.g., because the same
vocabulary item appears in multiple contexts, and/or because there are
different questions with the same answer). These will end up overlapping
in the same place in the visualisation, so let’s only keep unique
answers, choosing whichever fact has the most observations:
chars_to_remove <- "[^[:alnum:] ']" # Remove every character that isn't alphanumeric or an apostrophe
fact_rof[, answer_simplified := str_squish(str_replace_all(str_to_lower(answer), chars_to_remove, ""))]
fact_rof_nodup <- fact_rof[, .SD[n_obs == max(n_obs)], by = .(course, answer_simplified, answer_language)]
Embeddings
We’ll use a pretrained model to get embeddings for all the answers in
the set. With the text package we can get fastText
embeddings based on data from Common-Crawl and Wikipedia (see Grave et al., 2018
for details).
get_fasttext_embeddings <- function(sentences, model_path) {
# Save sentences to a temporary text file
temp_file <- tempfile(fileext = ".txt")
writeLines(sentences, temp_file)
# Save embeddings to a temporary text file
output_file <- tempfile()
# Run the fastText command-line interface to obtain embeddings
command <- paste("cd models && fastText/fasttext print-sentence-vectors", model_path, "<", temp_file, ">", output_file)
system(command)
# Get embeddings from file
embeddings <- read.table(output_file)
# Clean up temporary files
unlink(temp_file)
unlink(output_file)
return(embeddings)
}
# English
model_path_en <- here("models", "cc.en.300.bin")
embeddings_en <- cbind(fact_rof_nodup[answer_language == "EN"],
fact_rof_nodup[answer_language == "EN",
get_fasttext_embeddings(answer, model_path_en)])
# French
model_path_fr <- here("models", "cc.fr.300.bin")
embeddings_fr <- cbind(fact_rof_nodup[answer_language == "FR"],
fact_rof_nodup[answer_language == "FR",
get_fasttext_embeddings(answer, model_path_fr)])
Dimensionality reduction with UMAP
We now have 300-dimensional embeddings for each vocabulary item,
which is about 298 too many for a visualisation that we can understand.
The technique of Uniform Manifold Approximation and Projection (UMAP; McInnes, Healy, &
Melville, 2018) collapses these 300 dimensions down to 2. UMAP
attempts to preserve patterns in the high-dimensional space in its
lower-dimensional projection: it tries to keep items that are neighbours
close together, and also to maintain the global structure. It is
important to keep in mind that (distances along) the axes of the 2D plot
are in themselves not interpretable (in contrast to principle component
analysis).
UMAP has several parameters that define how it determines the
distance between items, the number of neighbours it considers around
each item, and the relative scaling of within- and between-cluster
space. There are no “correct” values to use and the best settings depend
on the nature of the data and the desired appearance of the end result
(e.g., should clusters be more or less clumped together). The settings I
have used below appear to be a good compromise that shows some local
structure while also leaving enough room for text labels within dense
clusters.
embeddings_en_umap <- umap(embeddings_en[,6:305],
n_components = 2,
metric = "cosine",
n_neighbors = 50,
min_dist = 0.5,
spread = 25
)
embeddings_en_umap_plot <- cbind(fact_rof_nodup[answer_language == "EN"],
as.data.frame(embeddings_en_umap, stringsAsFactors = FALSE))
embeddings_fr_umap <- umap(embeddings_fr[,6:305],
n_components = 2,
metric = "cosine",
n_neighbors = 50,
min_dist = 0.5,
spread = 15
)
embeddings_fr_umap_plot <- cbind(fact_rof_nodup[answer_language == "FR"],
as.data.frame(embeddings_fr_umap, stringsAsFactors = FALSE))
Visualisation
For the 2D visualisation, we want to prioritise plotting items with
high rate of forgetting. In case of overlapping labels, we’d rather see
the most difficult item.
setorder(embeddings_en_umap_plot, -rof)
setorder(embeddings_fr_umap_plot, -rof)
First, draw a version of the plot with a point for each vocabulary
item. The colour of the point shows the rate of forgetting.
p_en_umap <- ggplot(embeddings_en_umap_plot,
aes(x = V1, y = V2, colour = rof, label = answer)) +
geom_point(size = 1) +
scale_colour_scico(palette = "batlow") +
labs(colour = "Rate of\nforgetting") +
theme_void(base_size = 10) +
theme(panel.background = element_rect(fill = "black", colour = "black"),
legend.position = c(.95, .1),
legend.title = element_text(colour = "grey50", family = "Nunito", face = "bold"),
legend.text = element_text(colour = "grey50", family = "Nunito"))
p_en_umap

ggsave(p_en_umap,
filename = file.path("output", "embeddings_en_umap_points.png"),
width = 12,
height = 12,
dpi = 300,
device = png,
type = "cairo",
limitsize = FALSE)
p_fr_umap <- ggplot(embeddings_fr_umap_plot,
aes(x = -V1, y = V2, colour = rof, label = answer)) +
geom_point(size = 1) +
scale_colour_scico(palette = "batlow") +
labs(colour = "Rate of\nforgetting") +
theme_void(base_size = 10) +
theme(panel.background = element_rect(fill = "black", colour = "black"),
legend.position = c(.95, .1),
legend.title = element_text(colour = "grey50", family = "Nunito", face = "bold"),
legend.text = element_text(colour = "grey50", family = "Nunito"))
p_fr_umap

ggsave(p_fr_umap,
filename = file.path("output", "embeddings_fr_umap_points.png"),
width = 12,
height = 12,
dpi = 300,
device = png,
type = "cairo",
limitsize = FALSE)
We can see some really interesting structure in both languages. For
English, there appears to be a few regions on one side that have many of
the higer-RoF items. We also see some clusters. The French plot shows
stronger clustering, and also seems to show clear differences in average
RoF between different clusters. Interesting!
Now let’s make a bigger version of the plot with text labels on the
individual vocabulary items, so we can see which items they are.
p_en_umap_text <- ggplot(embeddings_en_umap_plot,
aes(x = V1, y = V2, colour = rof, label = answer)) +
geom_point(size = 3) +
geom_text(colour = "white",
size = ifelse(nchar(embeddings_en_umap_plot$answer) > 25, 1.5, 2),
nudge_y = .5, family = "Nunito", check_overlap = TRUE) +
geom_text(aes(colour = rof),
alpha = .75,
size = ifelse(nchar(embeddings_en_umap_plot$answer) > 25, 1.5, 2),
nudge_y = .5, family = "Nunito", check_overlap = TRUE) +
scale_colour_scico(palette = "batlow") +
labs(colour = "Rate of\nforgetting") +
theme_void(base_size = 10) +
theme(panel.background = element_rect(fill = "black", colour = "black"),
legend.position = c(.9625, .05),
legend.title = element_text(colour = "grey50", family = "Nunito", face = "bold"),
legend.text = element_text(colour = "grey50", family = "Nunito"))
ggsave(p_en_umap_text,
filename = file.path("output", "embeddings_en_umap_text.png"),
width = 33.1,
height = 33.1,
dpi = 300,
device = png,
type = "cairo",
limitsize = FALSE)
p_fr_umap_text <- ggplot(embeddings_fr_umap_plot,
aes(x = -V1, y = V2, colour = rof, label = answer)) +
geom_point(size = 3) +
geom_text(colour = "white",
size = ifelse(nchar(embeddings_fr_umap_plot$answer) > 25, 1.5, 2),
nudge_y = .45, family = "Nunito", check_overlap = TRUE) +
geom_text(aes(colour = rof),
alpha = .75,
size = ifelse(nchar(embeddings_fr_umap_plot$answer) > 25, 1.5, 2),
nudge_y = .45, family = "Nunito", check_overlap = TRUE) +
scale_colour_scico(palette = "batlow") +
labs(colour = "Rate of\nforgetting") +
theme_void(base_size = 10) +
theme(panel.background = element_rect(fill = "black", colour = "black"),
legend.position = c(.9625, .05),
legend.title = element_text(colour = "grey50", family = "Nunito", face = "bold"),
legend.text = element_text(colour = "grey50", family = "Nunito"))
ggsave(p_fr_umap_text,
filename = file.path("output", "embeddings_fr_umap_text.png"),
width = 33.1,
height = 33.1,
dpi = 300,
device = png,
type = "cairo",
limitsize = FALSE)
LS0tCnRpdGxlOiAiVmlzdWFsaXNpbmcgdGhlIG1lbW9yYWJpbGl0eSBvZiBzZWNvbmQtbGFuZ3VhZ2Ugdm9jYWJ1bGFyeSBpdGVtcyIKYXV0aG9yOiAiTWFhcnRlbiB2YW4gZGVyIFZlbGRlIgpkYXRlOiAiTGFzdCB1cGRhdGVkOiBgciBTeXMuRGF0ZSgpIGAiCm91dHB1dDoKICBnaXRodWJfZG9jdW1lbnQ6CiAgICB0b2M6IHllcwogICAgaHRtbF9wcmV2aWV3OiBubwogIGh0bWxfbm90ZWJvb2s6CiAgICBzbWFydDogbm8KICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwplZGl0b3Jfb3B0aW9uczogCiAgY2h1bmtfb3V0cHV0X3R5cGU6IGlubGluZQotLS0KCgojIFNldHVwCgpgYGB7cn0KbGlicmFyeShoZXJlKQpsaWJyYXJ5KGRhdGEudGFibGUpCmxpYnJhcnkoZnN0KQpsaWJyYXJ5KHRleHQpCmxpYnJhcnkoc3RyaW5ncikKbGlicmFyeShnZ3Bsb3QyKQpsaWJyYXJ5KHNjaWNvKQpsaWJyYXJ5KHBsb3RseSkKbGlicmFyeSh1d290KQpsaWJyYXJ5KGV4dHJhZm9udCkKCnNldC5zZWVkKDQyKQoKdGhlbWVfc2V0KHRoZW1lX2J3KGJhc2Vfc2l6ZSA9IDE2LCBiYXNlX2ZhbWlseSA9ICJzYW5zIikpCgojIFJ1biBvbmNlIHRvIGltcG9ydCB0aGUgTnVuaXRvIGZvbnQKIyBmb250X2ltcG9ydChwYXR0ZXJuID0gIk51bml0byIpCiMgbG9hZGZvbnRzKCkKYGBgCgojIERhdGEKCkxvYWQgcmF0ZSBvZiBmb3JnZXR0aW5nIHByZWRpY3Rpb25zIHBlciBmYWN0IChmcm9tIHRoZSAqRmFjdCogcHJlZGljdGlvbiBtZXRob2QgaW4gW2h0dHBzOi8vZG9pLm9yZy8xMC4zMTIzNC9vc2YuaW8vejN2dG5dKHZhbiBkZXIgVmVsZGUgZXQgYWwuLCAyMDIzKSksIGFzIHdlbGwgYXMgdGhlIHdvcmQgb3IgcGhyYXNlIGFzc29jaWF0ZWQgd2l0aCBlYWNoIGZhY3Q6CmBgYHtyfQojIEVuZ2xpc2gKcm9mX3ByZWRfZW4gPC0gcmVhZF9mc3QoaGVyZSgiZGF0YSIsICJwcmVkX2ZhY3RfU3RlcHBpbmdfU3RvbmVzLmZzdCIpLCBhcy5kYXRhLnRhYmxlID0gVFJVRSkKZmFjdHNfZW4gPC0gcmVhZF9mc3QoaGVyZSgiZGF0YSIsICJhbnN3ZXJzX1N0ZXBwaW5nX1N0b25lczIuZnN0IiksIGFzLmRhdGEudGFibGUgPSBUUlVFKQpzZXRuYW1lcyhmYWN0c19lbiwgImZhY3RfaWRfdW5pcV9tZXJnZWQiLCAiZmFjdF9pZCIpCgpmYWN0X3JvZl9lbiA8LSByb2ZfcHJlZF9lbltmYWN0c19lbiwgb24gPSAiZmFjdF9pZCJdCmZhY3Rfcm9mX2VuIDwtIGZhY3Rfcm9mX2VuWyFpcy5uYShtdSldCmZhY3Rfcm9mX2VuIDwtIGZhY3Rfcm9mX2VuWywgLihmYWN0X2lkLAogICAgICAgICAgICAgICAgICAgICAgICAgbl9vYnMgPSBrYXBwYSAtIDEsCiAgICAgICAgICAgICAgICAgICAgICAgICByb2YgPSBtdSwKICAgICAgICAgICAgICAgICAgICAgICAgIGFuc3dlciA9IGFuc3dlcildCgpmYWN0X3JvZl9lblssIGFuc3dlcl9sYW5ndWFnZSA6PSBpZmVsc2UodHN0cnNwbGl0KGZhY3RfaWQsICJfIiwgZml4ZWQgPSBUUlVFLCBrZWVwID0gM0wpW1sxXV0gPT0gIjEiLCAiTkwiLCAiRU4iKV0KZmFjdF9yb2ZfZW5bLCBjb3Vyc2UgOj0gIkVuZ2xpc2giXQoKIyBGcmVuY2gKcm9mX3ByZWRfZnIgPC0gcmVhZF9mc3QoZmlsZS5wYXRoKCJkYXRhIiwgInByZWRfZmFjdF9HcmFuZGVzX0xpZ25lcy5mc3QiKSwgYXMuZGF0YS50YWJsZSA9IFRSVUUpCmZhY3RzX2ZyIDwtIHJlYWRfZnN0KGZpbGUucGF0aCgiZGF0YSIsICJhbnN3ZXJzX0dyYW5kZXNfTGlnbmVzMi5mc3QiKSwgYXMuZGF0YS50YWJsZSA9IFRSVUUpCnNldG5hbWVzKGZhY3RzX2ZyLCAiZmFjdF9pZF91bmlxX21lcmdlZCIsICJmYWN0X2lkIikKCmZhY3Rfcm9mX2ZyIDwtIHJvZl9wcmVkX2ZyW2ZhY3RzX2ZyLCBvbiA9ICJmYWN0X2lkIl0KZmFjdF9yb2ZfZnIgPC0gZmFjdF9yb2ZfZnJbIWlzLm5hKG11KV0KZmFjdF9yb2ZfZnIgPC0gZmFjdF9yb2ZfZnJbLCAuKGZhY3RfaWQsCiAgICAgICAgICAgICAgICAgICAgICAgICBuX29icyA9IGthcHBhIC0gMSwKICAgICAgICAgICAgICAgICAgICAgICAgIHJvZiA9IG11LAogICAgICAgICAgICAgICAgICAgICAgICAgYW5zd2VyID0gYW5zd2VyKV0KCmZhY3Rfcm9mX2ZyWywgYW5zd2VyX2xhbmd1YWdlIDo9IGlmZWxzZSh0c3Ryc3BsaXQoZmFjdF9pZCwgIl8iLCBmaXhlZCA9IFRSVUUsIGtlZXAgPSAzTClbWzFdXSA9PSAiMSIsICJOTCIsICJGUiIpXQpmYWN0X3JvZl9mclssIGNvdXJzZSA6PSAiRnJlbmNoIl0KCiMgQ29tYmluZSBpbnRvIGEgc2luZ2xlIGRhdGEudGFibGUKZmFjdF9yb2YgPC0gcmJpbmQoZmFjdF9yb2ZfZW4sIGZhY3Rfcm9mX2ZyKQpgYGAKCgpTb21lIGFuc3dlcnMgb2NjdXIgYWNyb3NzIG11bHRpcGxlIGZhY3RzIChlLmcuLCBiZWNhdXNlIHRoZSBzYW1lIHZvY2FidWxhcnkgaXRlbSBhcHBlYXJzIGluIG11bHRpcGxlIGNvbnRleHRzLCBhbmQvb3IgYmVjYXVzZSB0aGVyZSBhcmUgZGlmZmVyZW50IHF1ZXN0aW9ucyB3aXRoIHRoZSBzYW1lIGFuc3dlcikuClRoZXNlIHdpbGwgZW5kIHVwIG92ZXJsYXBwaW5nIGluIHRoZSBzYW1lIHBsYWNlIGluIHRoZSB2aXN1YWxpc2F0aW9uLCBzbyBsZXQncyBvbmx5IGtlZXAgdW5pcXVlIGFuc3dlcnMsIGNob29zaW5nIHdoaWNoZXZlciBmYWN0IGhhcyB0aGUgbW9zdCBvYnNlcnZhdGlvbnM6CmBgYHtyfQpjaGFyc190b19yZW1vdmUgPC0gIlteWzphbG51bTpdICddIiAjIFJlbW92ZSBldmVyeSBjaGFyYWN0ZXIgdGhhdCBpc24ndCBhbHBoYW51bWVyaWMgb3IgYW4gYXBvc3Ryb3BoZQpmYWN0X3JvZlssIGFuc3dlcl9zaW1wbGlmaWVkIDo9IHN0cl9zcXVpc2goc3RyX3JlcGxhY2VfYWxsKHN0cl90b19sb3dlcihhbnN3ZXIpLCBjaGFyc190b19yZW1vdmUsICIiKSldCmZhY3Rfcm9mX25vZHVwIDwtIGZhY3Rfcm9mWywgLlNEW25fb2JzID09IG1heChuX29icyldLCBieSA9IC4oY291cnNlLCBhbnN3ZXJfc2ltcGxpZmllZCwgYW5zd2VyX2xhbmd1YWdlKV0KYGBgCgoKIyBFbWJlZGRpbmdzCgpXZSdsbCB1c2UgYSBwcmV0cmFpbmVkIG1vZGVsIHRvIGdldCBlbWJlZGRpbmdzIGZvciBhbGwgdGhlIGFuc3dlcnMgaW4gdGhlIHNldC4KV2l0aCB0aGUgYHRleHRgIHBhY2thZ2Ugd2UgY2FuIGdldCBbZmFzdFRleHQgZW1iZWRkaW5nc10oaHR0cHM6Ly9mYXN0dGV4dC5jYy9kb2NzL2VuL2NyYXdsLXZlY3RvcnMuaHRtbCkgYmFzZWQgb24gZGF0YSBmcm9tIENvbW1vbi1DcmF3bCBhbmQgV2lraXBlZGlhIChzZWUgW0dyYXZlIGV0IGFsLiwgMjAxOF0oaHR0cHM6Ly9kb2kub3JnLzEwLjQ4NTUwL2FyWGl2LjE4MDIuMDY4OTMpIGZvciBkZXRhaWxzKS4KCmBgYHtyfQpnZXRfZmFzdHRleHRfZW1iZWRkaW5ncyA8LSBmdW5jdGlvbihzZW50ZW5jZXMsIG1vZGVsX3BhdGgpIHsKICAjIFNhdmUgc2VudGVuY2VzIHRvIGEgdGVtcG9yYXJ5IHRleHQgZmlsZQogIHRlbXBfZmlsZSA8LSB0ZW1wZmlsZShmaWxlZXh0ID0gIi50eHQiKQogIHdyaXRlTGluZXMoc2VudGVuY2VzLCB0ZW1wX2ZpbGUpCiAgCiAgIyBTYXZlIGVtYmVkZGluZ3MgdG8gYSB0ZW1wb3JhcnkgdGV4dCBmaWxlCiAgb3V0cHV0X2ZpbGUgPC0gdGVtcGZpbGUoKQogIAogICMgUnVuIHRoZSBmYXN0VGV4dCBjb21tYW5kLWxpbmUgaW50ZXJmYWNlIHRvIG9idGFpbiBlbWJlZGRpbmdzCiAgY29tbWFuZCA8LSBwYXN0ZSgiY2QgbW9kZWxzICYmIGZhc3RUZXh0L2Zhc3R0ZXh0IHByaW50LXNlbnRlbmNlLXZlY3RvcnMiLCBtb2RlbF9wYXRoLCAiPCIsIHRlbXBfZmlsZSwgIj4iLCBvdXRwdXRfZmlsZSkKICBzeXN0ZW0oY29tbWFuZCkKICAKICAjIEdldCBlbWJlZGRpbmdzIGZyb20gZmlsZQogIGVtYmVkZGluZ3MgPC0gcmVhZC50YWJsZShvdXRwdXRfZmlsZSkKICAKICAjIENsZWFuIHVwIHRlbXBvcmFyeSBmaWxlcwogIHVubGluayh0ZW1wX2ZpbGUpCiAgdW5saW5rKG91dHB1dF9maWxlKQogIAogIHJldHVybihlbWJlZGRpbmdzKQp9CmBgYAoKYGBge3J9CiMgRW5nbGlzaAptb2RlbF9wYXRoX2VuIDwtIGhlcmUoIm1vZGVscyIsICJjYy5lbi4zMDAuYmluIikKZW1iZWRkaW5nc19lbiA8LSBjYmluZChmYWN0X3JvZl9ub2R1cFthbnN3ZXJfbGFuZ3VhZ2UgPT0gIkVOIl0sCiAgICAgICAgICAgICAgICAgICAgICAgZmFjdF9yb2Zfbm9kdXBbYW5zd2VyX2xhbmd1YWdlID09ICJFTiIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2V0X2Zhc3R0ZXh0X2VtYmVkZGluZ3MoYW5zd2VyLCBtb2RlbF9wYXRoX2VuKV0pCgojIEZyZW5jaAptb2RlbF9wYXRoX2ZyIDwtIGhlcmUoIm1vZGVscyIsICJjYy5mci4zMDAuYmluIikKZW1iZWRkaW5nc19mciA8LSBjYmluZChmYWN0X3JvZl9ub2R1cFthbnN3ZXJfbGFuZ3VhZ2UgPT0gIkZSIl0sCiAgICAgICAgICAgICAgICAgICAgICAgZmFjdF9yb2Zfbm9kdXBbYW5zd2VyX2xhbmd1YWdlID09ICJGUiIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZ2V0X2Zhc3R0ZXh0X2VtYmVkZGluZ3MoYW5zd2VyLCBtb2RlbF9wYXRoX2ZyKV0pCmBgYAoKCiMgRGltZW5zaW9uYWxpdHkgcmVkdWN0aW9uIHdpdGggVU1BUAoKV2Ugbm93IGhhdmUgMzAwLWRpbWVuc2lvbmFsIGVtYmVkZGluZ3MgZm9yIGVhY2ggdm9jYWJ1bGFyeSBpdGVtLCB3aGljaCBpcyBhYm91dCAyOTggdG9vIG1hbnkgZm9yIGEgdmlzdWFsaXNhdGlvbiB0aGF0IHdlIGNhbiB1bmRlcnN0YW5kLgpUaGUgdGVjaG5pcXVlIG9mIFVuaWZvcm0gTWFuaWZvbGQgQXBwcm94aW1hdGlvbiBhbmQgUHJvamVjdGlvbiAoVU1BUDsgW01jSW5uZXMsIEhlYWx5LCAmIE1lbHZpbGxlLCAyMDE4XShodHRwczovL2RvaS5vcmcvMTAuNDg1NTAvYXJYaXYuMTgwMi4wMzQyNikpIGNvbGxhcHNlcyB0aGVzZSAzMDAgZGltZW5zaW9ucyBkb3duIHRvIDIuClVNQVAgYXR0ZW1wdHMgdG8gcHJlc2VydmUgcGF0dGVybnMgaW4gdGhlIGhpZ2gtZGltZW5zaW9uYWwgc3BhY2UgaW4gaXRzIGxvd2VyLWRpbWVuc2lvbmFsIHByb2plY3Rpb246IGl0IHRyaWVzIHRvIGtlZXAgaXRlbXMgdGhhdCBhcmUgbmVpZ2hib3VycyBjbG9zZSB0b2dldGhlciwgYW5kIGFsc28gdG8gbWFpbnRhaW4gdGhlIGdsb2JhbCBzdHJ1Y3R1cmUuCkl0IGlzIGltcG9ydGFudCB0byBrZWVwIGluIG1pbmQgdGhhdCAoZGlzdGFuY2VzIGFsb25nKSB0aGUgYXhlcyBvZiB0aGUgMkQgcGxvdCBhcmUgaW4gdGhlbXNlbHZlcyBub3QgaW50ZXJwcmV0YWJsZSAoaW4gY29udHJhc3QgdG8gcHJpbmNpcGxlIGNvbXBvbmVudCBhbmFseXNpcykuCgpVTUFQIGhhcyBzZXZlcmFsIHBhcmFtZXRlcnMgdGhhdCBkZWZpbmUgaG93IGl0IGRldGVybWluZXMgdGhlIGRpc3RhbmNlIGJldHdlZW4gaXRlbXMsIHRoZSBudW1iZXIgb2YgbmVpZ2hib3VycyBpdCBjb25zaWRlcnMgYXJvdW5kIGVhY2ggaXRlbSwgYW5kIHRoZSByZWxhdGl2ZSBzY2FsaW5nIG9mIHdpdGhpbi0gYW5kIGJldHdlZW4tY2x1c3RlciBzcGFjZS4KVGhlcmUgYXJlIG5vICJjb3JyZWN0IiB2YWx1ZXMgdG8gdXNlIGFuZCB0aGUgYmVzdCBzZXR0aW5ncyBkZXBlbmQgb24gdGhlIG5hdHVyZSBvZiB0aGUgZGF0YSBhbmQgdGhlIGRlc2lyZWQgYXBwZWFyYW5jZSBvZiB0aGUgZW5kIHJlc3VsdCAoZS5nLiwgc2hvdWxkIGNsdXN0ZXJzIGJlIG1vcmUgb3IgbGVzcyBjbHVtcGVkIHRvZ2V0aGVyKS4KVGhlIHNldHRpbmdzIEkgaGF2ZSB1c2VkIGJlbG93IGFwcGVhciB0byBiZSBhIGdvb2QgY29tcHJvbWlzZSB0aGF0IHNob3dzIHNvbWUgbG9jYWwgc3RydWN0dXJlIHdoaWxlIGFsc28gbGVhdmluZyBlbm91Z2ggcm9vbSBmb3IgdGV4dCBsYWJlbHMgd2l0aGluIGRlbnNlIGNsdXN0ZXJzLgpgYGB7cn0KZW1iZWRkaW5nc19lbl91bWFwIDwtIHVtYXAoZW1iZWRkaW5nc19lblssNjozMDVdLAogICAgICAgICAgICAgICAgICAgICAgICAgICBuX2NvbXBvbmVudHMgPSAyLAogICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRyaWMgPSAiY29zaW5lIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgbl9uZWlnaGJvcnMgPSA1MCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgbWluX2Rpc3QgPSAwLjUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgIHNwcmVhZCA9IDI1CiAgICAgICAgICAgICAgICAgICAgICAgICAgICkKCgplbWJlZGRpbmdzX2VuX3VtYXBfcGxvdCA8LSBjYmluZChmYWN0X3JvZl9ub2R1cFthbnN3ZXJfbGFuZ3VhZ2UgPT0gIkVOIl0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFzLmRhdGEuZnJhbWUoZW1iZWRkaW5nc19lbl91bWFwLCBzdHJpbmdzQXNGYWN0b3JzID0gRkFMU0UpKQoKCmVtYmVkZGluZ3NfZnJfdW1hcCA8LSB1bWFwKGVtYmVkZGluZ3NfZnJbLDY6MzA1XSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgbl9jb21wb25lbnRzID0gMiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0cmljID0gImNvc2luZSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgIG5fbmVpZ2hib3JzID0gNTAsCiAgICAgICAgICAgICAgICAgICAgICAgICAgIG1pbl9kaXN0ID0gMC41LAogICAgICAgICAgICAgICAgICAgICAgICAgICBzcHJlYWQgPSAxNQogICAgICAgICAgICAgICAgICAgICAgICAgICApCgoKZW1iZWRkaW5nc19mcl91bWFwX3Bsb3QgPC0gY2JpbmQoZmFjdF9yb2Zfbm9kdXBbYW5zd2VyX2xhbmd1YWdlID09ICJGUiJdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhcy5kYXRhLmZyYW1lKGVtYmVkZGluZ3NfZnJfdW1hcCwgc3RyaW5nc0FzRmFjdG9ycyA9IEZBTFNFKSkKYGBgCgoKIyBWaXN1YWxpc2F0aW9uCgpGb3IgdGhlIDJEIHZpc3VhbGlzYXRpb24sIHdlIHdhbnQgdG8gcHJpb3JpdGlzZSBwbG90dGluZyBpdGVtcyB3aXRoIGhpZ2ggcmF0ZSBvZiBmb3JnZXR0aW5nLgpJbiBjYXNlIG9mIG92ZXJsYXBwaW5nIGxhYmVscywgd2UnZCByYXRoZXIgc2VlIHRoZSBtb3N0IGRpZmZpY3VsdCBpdGVtLgpgYGB7cn0Kc2V0b3JkZXIoZW1iZWRkaW5nc19lbl91bWFwX3Bsb3QsIC1yb2YpCnNldG9yZGVyKGVtYmVkZGluZ3NfZnJfdW1hcF9wbG90LCAtcm9mKQpgYGAKCkZpcnN0LCBkcmF3IGEgdmVyc2lvbiBvZiB0aGUgcGxvdCB3aXRoIGEgcG9pbnQgZm9yIGVhY2ggdm9jYWJ1bGFyeSBpdGVtLgpUaGUgY29sb3VyIG9mIHRoZSBwb2ludCBzaG93cyB0aGUgcmF0ZSBvZiBmb3JnZXR0aW5nLgpgYGB7ciBmaWcud2lkdGggPSAxMiwgZmlnLmhlaWdodCA9IDEyfQpwX2VuX3VtYXAgPC0gZ2dwbG90KGVtYmVkZGluZ3NfZW5fdW1hcF9wbG90LAogICAgICAgICAgICAgICAgICAgIGFlcyh4ID0gVjEsIHkgPSBWMiwgY29sb3VyID0gcm9mLCBsYWJlbCA9IGFuc3dlcikpICsKICBnZW9tX3BvaW50KHNpemUgPSAxKSArCiAgc2NhbGVfY29sb3VyX3NjaWNvKHBhbGV0dGUgPSAiYmF0bG93IikgKwogIGxhYnMoY29sb3VyID0gIlJhdGUgb2ZcbmZvcmdldHRpbmciKSArCiAgdGhlbWVfdm9pZChiYXNlX3NpemUgPSAxMCkgKwogIHRoZW1lKHBhbmVsLmJhY2tncm91bmQgPSBlbGVtZW50X3JlY3QoZmlsbCA9ICJibGFjayIsIGNvbG91ciA9ICJibGFjayIpLAogICAgICAgIGxlZ2VuZC5wb3NpdGlvbiA9IGMoLjk1LCAuMSksCiAgICAgICAgbGVnZW5kLnRpdGxlID0gZWxlbWVudF90ZXh0KGNvbG91ciA9ICJncmV5NTAiLCBmYW1pbHkgPSAiTnVuaXRvIiwgZmFjZSA9ICJib2xkIiksCiAgICAgICAgbGVnZW5kLnRleHQgPSBlbGVtZW50X3RleHQoY29sb3VyID0gImdyZXk1MCIsIGZhbWlseSA9ICJOdW5pdG8iKSkKCnBfZW5fdW1hcAoKZ2dzYXZlKHBfZW5fdW1hcCwKICAgICAgIGZpbGVuYW1lID0gZmlsZS5wYXRoKCJvdXRwdXQiLCAiZW1iZWRkaW5nc19lbl91bWFwX3BvaW50cy5wbmciKSwKICAgICAgIHdpZHRoID0gMTIsCiAgICAgICBoZWlnaHQgPSAxMiwKICAgICAgIGRwaSA9IDMwMCwKICAgICAgIGRldmljZSA9IHBuZywKICAgICAgIHR5cGUgPSAiY2Fpcm8iLAogICAgICAgbGltaXRzaXplID0gRkFMU0UpCgoKcF9mcl91bWFwIDwtIGdncGxvdChlbWJlZGRpbmdzX2ZyX3VtYXBfcGxvdCwKICAgICAgICAgICAgICAgICAgICBhZXMoeCA9IC1WMSwgeSA9IFYyLCBjb2xvdXIgPSByb2YsIGxhYmVsID0gYW5zd2VyKSkgKwogIGdlb21fcG9pbnQoc2l6ZSA9IDEpICsKICBzY2FsZV9jb2xvdXJfc2NpY28ocGFsZXR0ZSA9ICJiYXRsb3ciKSArCiAgbGFicyhjb2xvdXIgPSAiUmF0ZSBvZlxuZm9yZ2V0dGluZyIpICsKICB0aGVtZV92b2lkKGJhc2Vfc2l6ZSA9IDEwKSArCiAgdGhlbWUocGFuZWwuYmFja2dyb3VuZCA9IGVsZW1lbnRfcmVjdChmaWxsID0gImJsYWNrIiwgY29sb3VyID0gImJsYWNrIiksCiAgICAgICAgbGVnZW5kLnBvc2l0aW9uID0gYyguOTUsIC4xKSwKICAgICAgICBsZWdlbmQudGl0bGUgPSBlbGVtZW50X3RleHQoY29sb3VyID0gImdyZXk1MCIsIGZhbWlseSA9ICJOdW5pdG8iLCBmYWNlID0gImJvbGQiKSwKICAgICAgICBsZWdlbmQudGV4dCA9IGVsZW1lbnRfdGV4dChjb2xvdXIgPSAiZ3JleTUwIiwgZmFtaWx5ID0gIk51bml0byIpKQoKcF9mcl91bWFwCgpnZ3NhdmUocF9mcl91bWFwLAogICAgICAgZmlsZW5hbWUgPSBmaWxlLnBhdGgoIm91dHB1dCIsICJlbWJlZGRpbmdzX2ZyX3VtYXBfcG9pbnRzLnBuZyIpLAogICAgICAgd2lkdGggPSAxMiwKICAgICAgIGhlaWdodCA9IDEyLAogICAgICAgZHBpID0gMzAwLAogICAgICAgZGV2aWNlID0gcG5nLAogICAgICAgdHlwZSA9ICJjYWlybyIsCiAgICAgICBsaW1pdHNpemUgPSBGQUxTRSkKYGBgCldlIGNhbiBzZWUgc29tZSByZWFsbHkgaW50ZXJlc3Rpbmcgc3RydWN0dXJlIGluIGJvdGggbGFuZ3VhZ2VzLgpGb3IgRW5nbGlzaCwgdGhlcmUgYXBwZWFycyB0byBiZSBhIGZldyByZWdpb25zIG9uIG9uZSBzaWRlIHRoYXQgaGF2ZSBtYW55IG9mIHRoZSBoaWdlci1Sb0YgaXRlbXMuCldlIGFsc28gc2VlIHNvbWUgY2x1c3RlcnMuClRoZSBGcmVuY2ggcGxvdCBzaG93cyBzdHJvbmdlciBjbHVzdGVyaW5nLCBhbmQgYWxzbyBzZWVtcyB0byBzaG93IGNsZWFyIGRpZmZlcmVuY2VzIGluIGF2ZXJhZ2UgUm9GIGJldHdlZW4gZGlmZmVyZW50IGNsdXN0ZXJzLgpJbnRlcmVzdGluZyEKCk5vdyBsZXQncyBtYWtlIGEgYmlnZ2VyIHZlcnNpb24gb2YgdGhlIHBsb3Qgd2l0aCB0ZXh0IGxhYmVscyBvbiB0aGUgaW5kaXZpZHVhbCB2b2NhYnVsYXJ5IGl0ZW1zLCBzbyB3ZSBjYW4gc2VlIHdoaWNoIGl0ZW1zIHRoZXkgYXJlLgpgYGB7cn0KcF9lbl91bWFwX3RleHQgPC0gZ2dwbG90KGVtYmVkZGluZ3NfZW5fdW1hcF9wbG90LAogICAgICAgICAgICAgICAgICAgIGFlcyh4ID0gVjEsIHkgPSBWMiwgY29sb3VyID0gcm9mLCBsYWJlbCA9IGFuc3dlcikpICsKICBnZW9tX3BvaW50KHNpemUgPSAzKSArCiAgZ2VvbV90ZXh0KGNvbG91ciA9ICJ3aGl0ZSIsIAogICAgICAgICAgICBzaXplID0gaWZlbHNlKG5jaGFyKGVtYmVkZGluZ3NfZW5fdW1hcF9wbG90JGFuc3dlcikgPiAyNSwgMS41LCAyKSwKICAgICAgICAgICAgbnVkZ2VfeSA9IC41LCBmYW1pbHkgPSAiTnVuaXRvIiwgY2hlY2tfb3ZlcmxhcCA9IFRSVUUpICsKICBnZW9tX3RleHQoYWVzKGNvbG91ciA9IHJvZiksIAogICAgICAgICAgICBhbHBoYSA9IC43NSwgCiAgICAgICAgICAgIHNpemUgPSBpZmVsc2UobmNoYXIoZW1iZWRkaW5nc19lbl91bWFwX3Bsb3QkYW5zd2VyKSA+IDI1LCAxLjUsIDIpLAogICAgICAgICAgICBudWRnZV95ID0gLjUsIGZhbWlseSA9ICJOdW5pdG8iLCBjaGVja19vdmVybGFwID0gVFJVRSkgKwogIHNjYWxlX2NvbG91cl9zY2ljbyhwYWxldHRlID0gImJhdGxvdyIpICsKICBsYWJzKGNvbG91ciA9ICJSYXRlIG9mXG5mb3JnZXR0aW5nIikgKwogIHRoZW1lX3ZvaWQoYmFzZV9zaXplID0gMTApICsKICB0aGVtZShwYW5lbC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSAiYmxhY2siLCBjb2xvdXIgPSAiYmxhY2siKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSBjKC45NjI1LCAuMDUpLAogICAgICAgIGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfdGV4dChjb2xvdXIgPSAiZ3JleTUwIiwgZmFtaWx5ID0gIk51bml0byIsIGZhY2UgPSAiYm9sZCIpLAogICAgICAgIGxlZ2VuZC50ZXh0ID0gZWxlbWVudF90ZXh0KGNvbG91ciA9ICJncmV5NTAiLCBmYW1pbHkgPSAiTnVuaXRvIikpCgpnZ3NhdmUocF9lbl91bWFwX3RleHQsCiAgICAgICBmaWxlbmFtZSA9IGZpbGUucGF0aCgib3V0cHV0IiwgImVtYmVkZGluZ3NfZW5fdW1hcF90ZXh0LnBuZyIpLAogICAgICAgd2lkdGggPSAzMy4xLAogICAgICAgaGVpZ2h0ID0gMzMuMSwKICAgICAgIGRwaSA9IDMwMCwKICAgICAgIGRldmljZSA9IHBuZywKICAgICAgIHR5cGUgPSAiY2Fpcm8iLAogICAgICAgbGltaXRzaXplID0gRkFMU0UpCgoKcF9mcl91bWFwX3RleHQgPC0gZ2dwbG90KGVtYmVkZGluZ3NfZnJfdW1hcF9wbG90LAogICAgICAgICAgICAgICAgICAgIGFlcyh4ID0gLVYxLCB5ID0gVjIsIGNvbG91ciA9IHJvZiwgbGFiZWwgPSBhbnN3ZXIpKSArCiAgZ2VvbV9wb2ludChzaXplID0gMykgKwogIGdlb21fdGV4dChjb2xvdXIgPSAid2hpdGUiLCAKICAgICAgICAgICAgc2l6ZSA9IGlmZWxzZShuY2hhcihlbWJlZGRpbmdzX2ZyX3VtYXBfcGxvdCRhbnN3ZXIpID4gMjUsIDEuNSwgMiksCiAgICAgICAgICAgIG51ZGdlX3kgPSAuNDUsIGZhbWlseSA9ICJOdW5pdG8iLCBjaGVja19vdmVybGFwID0gVFJVRSkgKwogIGdlb21fdGV4dChhZXMoY29sb3VyID0gcm9mKSwgCiAgICAgICAgICAgIGFscGhhID0gLjc1LCAKICAgICAgICAgICAgc2l6ZSA9IGlmZWxzZShuY2hhcihlbWJlZGRpbmdzX2ZyX3VtYXBfcGxvdCRhbnN3ZXIpID4gMjUsIDEuNSwgMiksCiAgICAgICAgICAgIG51ZGdlX3kgPSAuNDUsIGZhbWlseSA9ICJOdW5pdG8iLCBjaGVja19vdmVybGFwID0gVFJVRSkgKwogIHNjYWxlX2NvbG91cl9zY2ljbyhwYWxldHRlID0gImJhdGxvdyIpICsKICBsYWJzKGNvbG91ciA9ICJSYXRlIG9mXG5mb3JnZXR0aW5nIikgKwogIHRoZW1lX3ZvaWQoYmFzZV9zaXplID0gMTApICsKICB0aGVtZShwYW5lbC5iYWNrZ3JvdW5kID0gZWxlbWVudF9yZWN0KGZpbGwgPSAiYmxhY2siLCBjb2xvdXIgPSAiYmxhY2siKSwKICAgICAgICBsZWdlbmQucG9zaXRpb24gPSBjKC45NjI1LCAuMDUpLAogICAgICAgIGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfdGV4dChjb2xvdXIgPSAiZ3JleTUwIiwgZmFtaWx5ID0gIk51bml0byIsIGZhY2UgPSAiYm9sZCIpLAogICAgICAgIGxlZ2VuZC50ZXh0ID0gZWxlbWVudF90ZXh0KGNvbG91ciA9ICJncmV5NTAiLCBmYW1pbHkgPSAiTnVuaXRvIikpCgoKCmdnc2F2ZShwX2ZyX3VtYXBfdGV4dCwKICAgICAgIGZpbGVuYW1lID0gZmlsZS5wYXRoKCJvdXRwdXQiLCAiZW1iZWRkaW5nc19mcl91bWFwX3RleHQucG5nIiksCiAgICAgICB3aWR0aCA9IDMzLjEsCiAgICAgICBoZWlnaHQgPSAzMy4xLAogICAgICAgZHBpID0gMzAwLAogICAgICAgZGV2aWNlID0gcG5nLAogICAgICAgdHlwZSA9ICJjYWlybyIsCiAgICAgICBsaW1pdHNpemUgPSBGQUxTRSkKYGBgCg==